Papers with similarity of neural network algorithm
SimSMoE: Toward Efficient Training Mixture of Experts via Solving Representational Collapse (2025.findings-naacl)
Copied to clipboard
| Challenge: | Sparse mixture of experts (SMoE) is an effective approach for scaling large language models with a constant computational cost. |
| Approach: | They propose a similarity-based sparse mixture of experts (SimSMoE) that guarantees a solution to address the representation collapse problem between experts given a fixed FLOPs budget. |
| Outcome: | The proposed method outperforms existing methods and outperformed other SMoE routing methods in performance. |